
<!doctype html>
<html lang="en" class="no-js">
  <head>
    
      <meta charset="utf-8">
      <meta name="viewport" content="width=device-width,initial-scale=1">
      
      
      
      <link rel="icon" href="../../../../static/images/favicon.png">
      <meta name="generator" content="mkdocs-1.3.0, mkdocs-material-8.2.8">
    
    
      
        <title>Prometheus + Grafana + node_exporter + Alertmanager 快速主机部署(生产建议使用k8s) - WL4G DOCS</title>
      
    
    
      <link rel="stylesheet" href="../../../../assets/stylesheets/main.644de097.min.css">
      
        
        <link rel="stylesheet" href="../../../../assets/stylesheets/palette.e6a45f82.min.css">
        
      
    
    
    
      
        
        
        <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
        <link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Roboto:300,300i,400,400i,700,700i%7CRoboto+Mono:400,400i,700,700i&display=fallback">
        <style>:root{--md-text-font:"Roboto";--md-code-font:"Roboto Mono"}</style>
      
    
    
      <link rel="stylesheet" href="../../../../static/css/util.css">
    
    <script>__md_scope=new URL("../../../..",location),__md_get=(e,_=localStorage,t=__md_scope)=>JSON.parse(_.getItem(t.pathname+"."+e)),__md_set=(e,_,t=localStorage,a=__md_scope)=>{try{t.setItem(a.pathname+"."+e,JSON.stringify(_))}catch(e){}}</script>
    
      

    
    
  </head>
  
  
    
    
      
    
    
    
    
    <body dir="ltr" data-md-color-scheme="default" data-md-color-primary="" data-md-color-accent="">
  
    
    
      <script>var palette=__md_get("__palette");if(palette&&"object"==typeof palette.color)for(var key of Object.keys(palette.color))document.body.setAttribute("data-md-color-"+key,palette.color[key])</script>
    
    <input class="md-toggle" data-md-toggle="drawer" type="checkbox" id="__drawer" autocomplete="off">
    <input class="md-toggle" data-md-toggle="search" type="checkbox" id="__search" autocomplete="off">
    <label class="md-overlay" for="__drawer"></label>
    <div data-md-component="skip">
      
        
        <a href="#prometheus-grafana-node_exporter-alertmanager-k8s" class="md-skip">
          Skip to content
        </a>
      
    </div>
    <div data-md-component="announce">
      
    </div>
    
      <div data-md-component="outdated" hidden>
        <aside class="md-banner md-banner--warning">
          
        </aside>
      </div>
    
    
      

<header class="md-header" data-md-component="header">
  <nav class="md-header__inner md-grid" aria-label="Header">
    <a href="../../../.." title="WL4G DOCS" class="md-header__button md-logo" aria-label="WL4G DOCS" data-md-component="logo">
      
  <img src="../../../../static/images/mylogo.jpeg" alt="logo">

    </a>
    <label class="md-header__button md-icon" for="__drawer">
      <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M3 6h18v2H3V6m0 5h18v2H3v-2m0 5h18v2H3v-2z"/></svg>
    </label>
    <div class="md-header__title" data-md-component="header-title">
      <div class="md-header__ellipsis">
        <div class="md-header__topic">
          <span class="md-ellipsis">
            WL4G DOCS
          </span>
        </div>
        <div class="md-header__topic" data-md-component="header-topic">
          <span class="md-ellipsis">
            
              Prometheus + Grafana + node_exporter + Alertmanager 快速主机部署(生产建议使用k8s)
            
          </span>
        </div>
      </div>
    </div>
    
      <form class="md-header__option" data-md-component="palette">
        
          
          
          <input class="md-option" data-md-color-media="(prefers-color-scheme: light)" data-md-color-scheme="default" data-md-color-primary="" data-md-color-accent=""  aria-label="Switch to dark mode"  type="radio" name="__palette" id="__palette_1">
          
            <label class="md-header__button md-icon" title="Switch to dark mode" for="__palette_2" hidden>
              <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M17 6H7c-3.31 0-6 2.69-6 6s2.69 6 6 6h10c3.31 0 6-2.69 6-6s-2.69-6-6-6zm0 10H7c-2.21 0-4-1.79-4-4s1.79-4 4-4h10c2.21 0 4 1.79 4 4s-1.79 4-4 4zM7 9c-1.66 0-3 1.34-3 3s1.34 3 3 3 3-1.34 3-3-1.34-3-3-3z"/></svg>
            </label>
          
        
          
          
          <input class="md-option" data-md-color-media="(prefers-color-scheme: dark)" data-md-color-scheme="slate" data-md-color-primary="" data-md-color-accent=""  aria-label="Switch to light mode"  type="radio" name="__palette" id="__palette_2">
          
            <label class="md-header__button md-icon" title="Switch to light mode" for="__palette_1" hidden>
              <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M17 7H7a5 5 0 0 0-5 5 5 5 0 0 0 5 5h10a5 5 0 0 0 5-5 5 5 0 0 0-5-5m0 8a3 3 0 0 1-3-3 3 3 0 0 1 3-3 3 3 0 0 1 3 3 3 3 0 0 1-3 3z"/></svg>
            </label>
          
        
      </form>
    
    
      <div class="md-header__option">
        <div class="md-select">
          
          <button class="md-header__button md-icon" aria-label="Select language">
            <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="m12.87 15.07-2.54-2.51.03-.03A17.52 17.52 0 0 0 14.07 6H17V4h-7V2H8v2H1v2h11.17C11.5 7.92 10.44 9.75 9 11.35 8.07 10.32 7.3 9.19 6.69 8h-2c.73 1.63 1.73 3.17 2.98 4.56l-5.09 5.02L4 19l5-5 3.11 3.11.76-2.04M18.5 10h-2L12 22h2l1.12-3h4.75L21 22h2l-4.5-12m-2.62 7 1.62-4.33L19.12 17h-3.24z"/></svg>
          </button>
          <div class="md-select__inner">
            <ul class="md-select__list">
              
                <li class="md-select__item">
                  <a href="/en/" hreflang="en" class="md-select__link">
                    English
                  </a>
                </li>
                
                <li class="md-select__item">
                  <a href="/zh/" hreflang="zh" class="md-select__link">
                    简体中文
                  </a>
                </li>
                
            </ul>
          </div>
        </div>
      </div>
    
    
      <label class="md-header__button md-icon" for="__search">
        <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M9.5 3A6.5 6.5 0 0 1 16 9.5c0 1.61-.59 3.09-1.56 4.23l.27.27h.79l5 5-1.5 1.5-5-5v-.79l-.27-.27A6.516 6.516 0 0 1 9.5 16 6.5 6.5 0 0 1 3 9.5 6.5 6.5 0 0 1 9.5 3m0 2C7 5 5 7 5 9.5S7 14 9.5 14 14 12 14 9.5 12 5 9.5 5z"/></svg>
      </label>
      <div class="md-search" data-md-component="search" role="dialog">
  <label class="md-search__overlay" for="__search"></label>
  <div class="md-search__inner" role="search">
    <form class="md-search__form" name="search">
      <input type="text" class="md-search__input" name="query" aria-label="Search" placeholder="Search" autocapitalize="off" autocorrect="off" autocomplete="off" spellcheck="false" data-md-component="search-query" required>
      <label class="md-search__icon md-icon" for="__search">
        <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M9.5 3A6.5 6.5 0 0 1 16 9.5c0 1.61-.59 3.09-1.56 4.23l.27.27h.79l5 5-1.5 1.5-5-5v-.79l-.27-.27A6.516 6.516 0 0 1 9.5 16 6.5 6.5 0 0 1 3 9.5 6.5 6.5 0 0 1 9.5 3m0 2C7 5 5 7 5 9.5S7 14 9.5 14 14 12 14 9.5 12 5 9.5 5z"/></svg>
        <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M20 11v2H8l5.5 5.5-1.42 1.42L4.16 12l7.92-7.92L13.5 5.5 8 11h12z"/></svg>
      </label>
      <nav class="md-search__options" aria-label="Search">
        
        <button type="reset" class="md-search__icon md-icon" aria-label="Clear" tabindex="-1">
          <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M19 6.41 17.59 5 12 10.59 6.41 5 5 6.41 10.59 12 5 17.59 6.41 19 12 13.41 17.59 19 19 17.59 13.41 12 19 6.41z"/></svg>
        </button>
      </nav>
      
        <div class="md-search__suggest" data-md-component="search-suggest"></div>
      
    </form>
    <div class="md-search__output">
      <div class="md-search__scrollwrap" data-md-scrollfix>
        <div class="md-search-result" data-md-component="search-result">
          <div class="md-search-result__meta">
            Initializing search
          </div>
          <ol class="md-search-result__list"></ol>
        </div>
      </div>
    </div>
  </div>
</div>
    
    
  </nav>
  
</header>
    
    <div class="md-container" data-md-component="container">
      
      
        
          
            
<nav class="md-tabs" aria-label="Tabs" data-md-component="tabs">
  <div class="md-tabs__inner md-grid">
    <ul class="md-tabs__list">
      
        
  
  


  
  
  
    <li class="md-tabs__item">
      <a href="../../../.." class="md-tabs__link">
        Getting Started
      </a>
    </li>
  

      
    </ul>
  </div>
</nav>
          
        
      
      <main class="md-main" data-md-component="main">
        <div class="md-main__inner md-grid">
          
            
              
              <div class="md-sidebar md-sidebar--primary" data-md-component="sidebar" data-md-type="navigation" >
                <div class="md-sidebar__scrollwrap">
                  <div class="md-sidebar__inner">
                    

  


  

<nav class="md-nav md-nav--primary md-nav--lifted md-nav--integrated" aria-label="Navigation" data-md-level="0">
  <label class="md-nav__title" for="__drawer">
    <a href="../../../.." title="WL4G DOCS" class="md-nav__button md-logo" aria-label="WL4G DOCS" data-md-component="logo">
      
  <img src="../../../../static/images/mylogo.jpeg" alt="logo">

    </a>
    WL4G DOCS
  </label>
  
  <ul class="md-nav__list" data-md-scrollfix>
    
      
      
      

  
  
  
    
    <li class="md-nav__item md-nav__item--nested">
      
      
        <input class="md-nav__toggle md-toggle" data-md-toggle="__nav_1" data-md-state="indeterminate" type="checkbox" id="__nav_1" checked>
      
      
      
      
        <label class="md-nav__link" for="__nav_1">
          Getting Started
          <span class="md-nav__icon md-icon"></span>
        </label>
      
      <nav class="md-nav" aria-label="Getting Started" data-md-level="1">
        <label class="md-nav__title" for="__nav_1">
          <span class="md-nav__icon md-icon"></span>
          Getting Started
        </label>
        <ul class="md-nav__list" data-md-scrollfix>
          
            
              
  
  
  
    <li class="md-nav__item">
      <a href="../../../.." class="md-nav__link">
        Introduction
      </a>
    </li>
  

            
          
            
              
  
  
  
    <li class="md-nav__item">
      <a href="../../../../ABOUT_CN/" class="md-nav__link">
        About
      </a>
    </li>
  

            
          
        </ul>
      </nav>
    </li>
  

    
  </ul>
</nav>
                  </div>
                </div>
              </div>
            
            
          
          <div class="md-content" data-md-component="content">
            <article class="md-content__inner md-typeset">
              
                


<h1 id="prometheus-grafana-node_exporter-alertmanager-k8s">Prometheus + Grafana + node_exporter + Alertmanager 快速主机部署(生产建议使用k8s)<a class="headerlink" href="#prometheus-grafana-node_exporter-alertmanager-k8s" title="Permanent link">&para;</a></h1>
<ul>
<li>Architectures</li>
</ul>
<table>
<thead>
<tr>
<th>IP</th>
<th>Servers</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td>10.111.178.62</td>
<td>Prometheus / Grafana / Alertmanager</td>
<td>为了简单均采用单点 docker bridge port-forward 模式</td>
</tr>
<tr>
<td>10.111.0.111</td>
<td>node_exporter</td>
<td>为了简单采用 systemd 管理(生产采用 k8s DaemonSet 方式)</td>
</tr>
<tr>
<td>10.111.0.112</td>
<td>node_exporter</td>
<td>为了简单采用 systemd 管理(生产采用 k8s DaemonSet 方式)</td>
</tr>
</tbody>
</table>
<h2 id="1-prometheus">1. 部署 <a href="https://github.com/wl4g-collect/prometheus">Prometheus</a><a class="headerlink" href="#1-prometheus" title="Permanent link">&para;</a></h2>
<p><details style="cursor:pointer;;padding-left:2%">
<summary style="color:#007acc;">1.1 [展开] Installation & configure prometheus</summary>
<code><pre>
# @see: https://hub.docker.com/r/prom/prometheus/tags
docker pull docker.io/prom/prometheus:v2.30.3

sudo mkdir -p /etc/prometheus

# ----- prometheus.yml -----
sudo cat <<-'EOF'>/etc/prometheus/prometheus.yml
# my global config
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
           - 10.111.178.63:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  - "emqx-alert-rules.yml"
  - "haproxy-alert-rules.yml"
  - "jvm-alert-rules.yml"
  - "kafka-alert-rules.yml"
  - "node-alert-rules.yml"
  - "prometheus-alert-rules.yml"
  - "redis-alert-rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: "prometheus"
    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.
    static_configs:
      - targets: ["localhost:9090"]
  - job_name: node
    static_configs:
    - targets: ['10.111.0.111:9100']
      labels:
        instance: collect-node1
    - targets: ['10.111.0.112:9100']
      labels:
        instance: collect-node2
    - targets: ['10.111.178.79:9100']
      labels:
        instance: web-node2
    - targets: ['10.111.178.78:9100']
      labels:
        instance: web-node3
    - targets: ['10.111.178.60:9100']
      labels:
        instance: compute-node1
    - targets: ['10.111.0.11:9100']
      labels:
        instance: kafka-follower1
    - targets: ['10.111.0.12:9100']
      labels:
        instance: kafka-follower2
    - targets: ['10.29.69.150:9100']
      labels:
        instance: kafka-leader1
    - targets: ['10.111.178.58:9100']
      labels:
        instance: sink-node1
    - targets: ['10.111.178.57:9100']
      labels:
        instance: sink-node2
    - targets: ['10.111.178.70:9100']
      labels:
        instance: repo-node1
    - targets: ['10.111.178.63:9100']
      labels:
        instance: devops-node1
    - targets: ['10.111.178.72:9100']
      labels:
        instance: emr-header-1.cluster-125585
    - targets: ['10.111.178.73:9100']
      labels:
        instance: emr-header-2.cluster-125585
    - targets: ['10.111.178.71:9100']
      labels:
        instance: emr-worker-1.cluster-125585
    - targets: ['10.111.178.74:9100']
      labels:
        instance: emr-worker-2.cluster-125585
    - targets: ['10.111.178.75:9100']
      labels:
        instance: emr-worker-3.cluster-125585
    - targets: ['10.111.178.76:9100']
      labels:
        instance: emr-worker-4.cluster-125585
  - job_name: haproxy
    static_configs:
    - targets: ['10.111.0.111:9101']
      labels:
        instance: collect-node1:9101
    - targets: ['10.111.0.112:9101']
      labels:
        instance: collect-node2:9101
  - job_name: iam-gateway
    metrics_path: /actuator/prometheus
    static_configs:
    - targets: ['10.111.178.79:18086']
      labels:
        instance: web-node2:18086
    - targets: ['10.111.178.78:18086']
      labels:
        instance: web-node3:18086
  - job_name: kafka
    static_configs:
    - targets: ['10.111.0.11:10105']
      labels:
        instance: kafka-follower1:10105
    - targets: ['10.111.0.12:10105']
      labels:
        instance: kafka-follower2:10105
    - targets: ['10.29.69.150:10105']
      labels:
        instance: kafka-leader1:10105
  - job_name: emq
    static_configs:
    - targets: ['10.111.0.111:9091']
      labels:
        instance: collect-node1:9091
    - targets: ['10.111.0.112:9091']
      labels:
        instance: collect-node2:9091
EOF
</pre></code>
</details></p>

<ul>
<li>
<p>更多告警规则参见: <a href="https://github.com/wl4g/prometheus-integration/tree/master/prometheus">github.com/wl4g/prometheus-integration/tree/master/prometheus</a></p>
</li>
<li>
<p>1.2 启动 prometheus server</p>
</li>
</ul>
<div class="highlight"><pre><span></span><code><a id="__codelineno-0-1" name="__codelineno-0-1"></a><a href="#__codelineno-0-1"><span class="linenos" data-linenos=" 1 "></span></a>mkdir -p /mnt/disk1/prometheus<span class="p">;</span> chmod -R <span class="m">777</span> /mnt/disk1/prometheus
<a id="__codelineno-0-2" name="__codelineno-0-2"></a><a href="#__codelineno-0-2"><span class="linenos" data-linenos=" 2 "></span></a>docker run -d --name<span class="o">=</span>prometheus1 -p <span class="m">9090</span>:9090 <span class="se">\</span>
<a id="__codelineno-0-3" name="__codelineno-0-3"></a><a href="#__codelineno-0-3"><span class="linenos" data-linenos=" 3 "></span></a>-v /etc/prometheus/:/etc/prometheus/ <span class="se">\</span>
<a id="__codelineno-0-4" name="__codelineno-0-4"></a><a href="#__codelineno-0-4"><span class="linenos" data-linenos=" 4 "></span></a>-v /mnt/disk1/prometheus/:/prometheus/ <span class="se">\</span>
<a id="__codelineno-0-5" name="__codelineno-0-5"></a><a href="#__codelineno-0-5"><span class="linenos" data-linenos=" 5 "></span></a>--network host <span class="se">\</span>
<a id="__codelineno-0-6" name="__codelineno-0-6"></a><a href="#__codelineno-0-6"><span class="linenos" data-linenos=" 6 "></span></a>--restart<span class="o">=</span>always prom/prometheus:v2.30.3 <span class="se">\</span>
<a id="__codelineno-0-7" name="__codelineno-0-7"></a><a href="#__codelineno-0-7"><span class="linenos" data-linenos=" 7 "></span></a>--config.file<span class="o">=</span>/etc/prometheus/prometheus.yml <span class="se">\</span>
<a id="__codelineno-0-8" name="__codelineno-0-8"></a><a href="#__codelineno-0-8"><span class="linenos" data-linenos=" 8 "></span></a>--storage.tsdb.path<span class="o">=</span>/prometheus <span class="se">\</span>
<a id="__codelineno-0-9" name="__codelineno-0-9"></a><a href="#__codelineno-0-9"><span class="linenos" data-linenos=" 9 "></span></a>--web.console.libraries<span class="o">=</span>/usr/share/prometheus/console_libraries <span class="se">\</span>
<a id="__codelineno-0-10" name="__codelineno-0-10"></a><a href="#__codelineno-0-10"><span class="linenos" data-linenos="10 "></span></a>--web.console.templates<span class="o">=</span>/usr/share/prometheus/consoles <span class="se">\</span>
<a id="__codelineno-0-11" name="__codelineno-0-11"></a><a href="#__codelineno-0-11"><span class="linenos" data-linenos="11 "></span></a>--storage.tsdb.retention<span class="o">=</span>180d <span class="se">\</span>
<a id="__codelineno-0-12" name="__codelineno-0-12"></a><a href="#__codelineno-0-12"><span class="linenos" data-linenos="12 "></span></a>--web.listen-address<span class="o">=</span><span class="s2">&quot;0.0.0.0:9090&quot;</span> <span class="se">\</span>
<a id="__codelineno-0-13" name="__codelineno-0-13"></a><a href="#__codelineno-0-13"><span class="linenos" data-linenos="13 "></span></a>--web.enable-admin-api
<a id="__codelineno-0-14" name="__codelineno-0-14"></a><a href="#__codelineno-0-14"><span class="linenos" data-linenos="14 "></span></a>
<a id="__codelineno-0-15" name="__codelineno-0-15"></a><a href="#__codelineno-0-15"><span class="linenos" data-linenos="15 "></span></a><span class="c1"># 【可选】临时测试启动，停止自动销毁（可观察创建过程日志 stdout 输出）</span>
<a id="__codelineno-0-16" name="__codelineno-0-16"></a><a href="#__codelineno-0-16"><span class="linenos" data-linenos="16 "></span></a><span class="c1">#docker run --rm --name=prometheus1 -p 9090:9090 \</span>
<a id="__codelineno-0-17" name="__codelineno-0-17"></a><a href="#__codelineno-0-17"><span class="linenos" data-linenos="17 "></span></a>-v /etc/prometheus/:/etc/prometheus/ <span class="se">\</span>
<a id="__codelineno-0-18" name="__codelineno-0-18"></a><a href="#__codelineno-0-18"><span class="linenos" data-linenos="18 "></span></a>-v /mnt/disk1/prometheus/:/prometheus/ <span class="se">\</span>
<a id="__codelineno-0-19" name="__codelineno-0-19"></a><a href="#__codelineno-0-19"><span class="linenos" data-linenos="19 "></span></a>prom/prometheus:v2.30.3
</code></pre></div>
<p>以上除了 <code>--storage.tsdb.retention=180d --web.enable-admin-api</code> 其他均为 prometheus 官方容器默认启动参数，由于 <code>docker run ...</code> 时指定的 COMMAND 和 ARGS 会覆盖默认启动命令，因此必须全部指定。参见官方文档: <a href="https://docs.docker.com/engine/reference/run/#cmd-default-command-or-options">docker-run#cmd-default-command-or-options</a></p>
<ul>
<li>控制台界面: http://10.111.178.62:9090/</li>
</ul>
<h2 id="2-grafana">2. 部署 <a href="https://grafana.com/oss/prometheus/exporters/node-exporter/?tab=dashboards#step-1-setting-up-node-exporter">Grafana</a><a class="headerlink" href="#2-grafana" title="Permanent link">&para;</a></h2>
<p><details style="cursor:pointer;;padding-left:2%">
<summary style="color:#007acc;">[展开] Installation grafana</summary>
<code><pre>
# see: https://hub.docker.com/r/grafana/grafana/tags
docker pull docker.io/grafana/grafana:8.2.2

mkdir -p /mnt/disk1/grafana; chmod -R 777 /mnt/disk1/grafana
docker run -d --name=grafana1 -p 3000:3000 --network host --restart=always -v /mnt/disk1/grafana:/var/lib/grafana docker.io/grafana/grafana:8.2.2
</pre></code>
</details></p>

<ul>
<li>控制台界面: http://10.111.178.62:3000/  <code>admin/初始密码为任意非空字符串</code>，建议首次登录进去后立即修改密码</li>
<li>添加 Prometheus 数据源
<img alt="" src="https://blogs.wl4g.com:443/wp-content/uploads/2021/10/prom-grafana1.png" /></li>
</ul>
<h2 id="3-node_exporter">3. 部署 <a href="https://github.com/wl4g-collect/node_exporter">node_exporter</a><a class="headerlink" href="#3-node_exporter" title="Permanent link">&para;</a></h2>
<ul>
<li>官方推荐更多 exporter: <a href="https://prometheus.io/docs/instrumenting/exporters/#exporters-and-integrations">https://prometheus.io/docs/instrumenting/exporters/</a></li>
</ul>
<h3 id="31">3.1 安装及配置<a class="headerlink" href="#31" title="Permanent link">&para;</a></h3>
<ul>
<li>下载安装</li>
</ul>
<p><details style="cursor:pointer;;padding-left:2%">
<summary style="color:#007acc;">[展开] Installation node_exporter</summary>
<code><pre>
# ------ Downloading installation -------
sudo cd /tmp
sudo curl -OL https://github.91chifun.workers.dev/https://github.com//prometheus/node_exporter/releases/download/v1.2.2/node_exporter-1.2.2.linux-amd64.tar.gz
sudo tar -xf node_exporter*
sudo cp node_exporter*/node_exporter /usr/sbin/

# ------ Make configuration -------
sudo cat <<-'EOF' >/etc/sysconfig/node_exporter
OPTIONS="--collector.systemd \
--collector.mdadm \
--collector.tcpstat \
--collector.processes \
--web.listen-address=:9100 \
--web.telemetry-path=/metrics"
EOF
</pre></code>
</details></p>

<ul>
<li>配置服务脚本(推荐，适用于如 CentOS 7.x)</li>
</ul>
<p><details style="cursor:pointer;;padding-left:2%">
<summary style="color:#007acc;">[展开] Configure node_exporter(systemd)</summary>
<code><pre>
sudo cat <<-'EOF' >/etc/systemd/system/node_exporter.service
[Unit]
Description=Node Exporter

[Service]
User=root
Group=root
EnvironmentFile=-/etc/sysconfig/node_exporter
ExecStart=/usr/sbin/node_exporter $OPTIONS
ExecReload=/bin/kill -s HUP
StandardOutput=journal
StandardError=journal
LimitNOFILE=64
LimitNPROC=64
LimitCORE=infinity
TimeoutStartSec=10
TimeoutSec=300
Restart=always

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable node_exporter
sudo systemctl restart node_exporter
sudo systemctl status node_exporter
sudo journalctl -u node_exporter -f

# 测试访问
curl http://localhost:9100/metrics
</pre></code>
</details></p>

<ul>
<li>配置服务脚本(可选，适用于如 CentOS 6.x)</li>
</ul>
<p><details style="cursor:pointer;;padding-left:2%">
<summary style="color:#007acc;">[展开] Configure node_exporter(init.d)</summary>
<code><pre>
sudo cat <<-'EOF' >/etc/init.d/node_exporter.sh
#!/bin/bash
#/*
# * Copyright 2017 ~ 2025 the original author or authors. <Wanglsir@gmail.com, 983708408@qq.com>
# *
# * Licensed under the Apache License, Version 2.0 (the "License");
# * you may not use this file except in compliance with the License.
# * You may obtain a copy of the License at
# *
# *      http://www.apache.org/licenses/LICENSE-2.0
# *
# * Unless required by applicable law or agreed to in writing, software
# * distributed under the License is distributed on an "AS IS" BASIS,
# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# * See the License for the specific language governing permissions and
# * limitations under the License.
# */

[ -f /etc/sysconfig/network ] && . /etc/sysconfig/network
[ "$NETWORKING" = "no" ] && exit 0

# Environment definition.
binFile="$(command -v node_exporter)"
if [ -f /etc/sysconfig/node_exporter ]; then
  . /etc/sysconfig/node_exporter
  export NODE_EXPORTER_OPTIONS=$OPTIONS
fi

function start() {
  local pids=$(getPids)
  if [ -z "$pids" ]; then
    nohup $binFile $NODE_EXPORTER_OPTIONS > "/var/log/node_exporter.stdout" 2>&1 &
    echo -n "Starting node exporter ..."
    while true
    do
      pids=$(getPids)
      if [ "$pids" == "" ]; then
        echo -n ".";
        sleep 0.8;
      else
        echo $pids >"/run/node_exporter.pid"
        break
      fi
    done
    echo -e "\nStarted node_exporter on "$pids
  else
    echo "Node exporter process is running "$pids
  fi
}

function stop() {
  local pids=$(getPids)
  if [ -z "$pids" ]; then
    echo "Node exporter not running!"
  else
    echo -n "Stopping node_exporter for $pids ..."
    kill -s TERM $pids
    while true
    do
      pids=$(getPids)
      if [ "$pids" == "" ]; then
        \rm -f /run/node_exporter.pid
        break
      else
        echo -n ".";
        sleep 0.8;
      fi
    done
    echo -e "\nStopped node_exporter !"
  fi
}

function status() {
  ps -ef | grep -v grep | grep $binFile
}

function getPids() {
  local pids=$(ps ax | grep -i "$binFile" | grep -v grep | awk '{print $1}')
  echo $pids # Output execution result value.
  return 0 # Return the execution result code.
}

# --- Main call. ---
CMD=$1
case $CMD in
  status)
    status
    ;;
  start)
    start
    ;;
  stop)
    stop
    ;;
  restart)
    stop
    start
    ;;
    *)
  echo $"Usage: {start|stop|restart|status}"
  exit 2
esac
EOF

chmod +x /etc/init.d/node_exporter.sh
/etc/init.d/node_exporter.sh restart
/etc/init.d/node_exporter.sh status

# 测试访问
curl http://localhost:9100/metrics
</pre></code>
</details></p>

<h3 id="32-node_exporter">3.2 添加仪表盘 (node_exporter)<a class="headerlink" href="#32-node_exporter" title="Permanent link">&para;</a></h3>
<ul>
<li>推荐简单 Grafana ID: <a href="https://grafana.com/grafana/dashboards/13978">13978</a></li>
<li>推荐热门 Grafana ID: <a href="https://grafana.com/grafana/dashboards/8919">8919</a> &nbsp;&nbsp;-&nbsp;&nbsp; 1 Node Exporter for Prometheus Dashboard CN</li>
<li>推荐热门 Grafana ID: <a href="https://grafana.com/grafana/dashboards/11074">11074</a> &nbsp;&nbsp;-&nbsp;&nbsp; 1 Node Exporter for Prometheus Dashboard EN</li>
<li>推荐热门 Grafana ID: <a href="https://grafana.com/grafana/dashboards/15172">15172</a> &nbsp;&nbsp;-&nbsp;&nbsp; 1 Node Enhanced version based on 11074:Exporter for Prometheus Dashboard EN, or download1: <a href="https://gitee.com/wl4g/blogs/raw/master/docs/articles/operation/prometheus-grafana-node_exporter-host-deployment/data/node_exporter-15172-rev6.json">node_exporter-15172-rev6.json</a>, download2: <a href="http://pkg.wl4g.com/software/grafana/node_exporter-15172-rev6.json">node_exporter-15172-rev6.json</a></li>
<li>推荐热门 Grafana ID: <a href="https://grafana.com/grafana/dashboards/1860">1860</a> &nbsp;&nbsp;-&nbsp;&nbsp; 1 Node Exporter Full</li>
<li>推荐热门 Grafana ID: <a href="https://grafana.com/grafana/dashboards/315">315</a> &nbsp;&nbsp;-&nbsp;&nbsp; Kubernetes cluster monitoring (via Prometheus)</li>
<li>更多优秀 Dashboard: <a href="https://grafana.com/grafana/dashboards">https://grafana.com/grafana/dashboards</a></li>
</ul>
<h3 id="33">3.3 效果图<a class="headerlink" href="#33" title="Permanent link">&para;</a></h3>
<ul>
<li>
<p><a href="https://grafana.com/grafana/dashboards/13978">13978</a>
<img alt="" src="https://blogs.wl4g.com:443/wp-content/uploads/2021/10/prom-grafana.png" /></p>
</li>
<li>
<p><a href="https://grafana.com/grafana/dashboards/8919">8919</a>
<img alt="" src="https://blogs.wl4g.com:443/wp-content/uploads/2021/10/prom-grafana1-1.png" /></p>
</li>
<li>
<p><a href="https://grafana.com/grafana/dashboards/1860">1860</a>
<img alt="" src="https://blogs.wl4g.com:443/wp-content/uploads/2021/10/prom-grafana1-2.png" /></p>
</li>
</ul>
<h3 id="34">3.4 常见重要生产环境指标<a class="headerlink" href="#34" title="Permanent link">&para;</a></h3>
<table>
<thead>
<tr>
<th>指标名</th>
<th>启用参数</th>
<th>默认启用</th>
<th>参考意义</th>
</tr>
</thead>
<tbody>
<tr>
<td>node_memory_MemFree_bytes, node_memory_Buffers_bytes, node_memory_Cached_bytes, node_memory_SwapCached_bytes, node_memory_VmallocUsed_bytes</td>
<td>--collector.meminfo</td>
<td>是</td>
<td>当前内存使用情况, kernel 的内存申请分配机制复杂</td>
</tr>
<tr>
<td>node_netstat_Tcp_CurrEstab</td>
<td>--collector.netstat</td>
<td>是</td>
<td>高并发时响应延迟或拒绝连接连接数排查</td>
</tr>
<tr>
<td>node_tcp_connection_states</td>
<td>--collector.tcpstat</td>
<td>否</td>
<td>高并发时响应延迟或拒绝连接连接数排查</td>
</tr>
<tr>
<td>node_processes_threads</td>
<td>--collector.processes</td>
<td>否</td>
<td>【重要】通常都关心 OOM 等 mem问题，较少关心系统的 Thread 情况，亲经历 netty 服务器高并时耗尽 <a href="https://github1s.com/torvalds/linux/blob/v3.10/include/linux/threads.h">pid_max</a>(如 CentOS 7.9 默认 32768) ，导致 ssh 无法登陆及 http 服务无法响应 <a href="https://blog.csdn.net/woshizhangliang999/article/details/66473884">fork: Cannot allocate memory</a></td>
</tr>
</tbody>
</table>
<h3 id="35">3.5 指标类型及应用场景：<a class="headerlink" href="#35" title="Permanent link">&para;</a></h3>
<ul>
<li>Counter：收集事件次数等单调递增的数据</li>
<li>Gauge：收集当前的状态，比如数据库连接数</li>
<li>Histogram：收集随机正态分布数据，比如响应延迟</li>
<li>Summary：收集随机正态分布数据，和 Histogram 是类似的</li>
</ul>
<h2 id="4-alertmanager">4. 部署 <a href="https://github.com/wl4g-collect/alertmanager">alertmanager</a><a class="headerlink" href="#4-alertmanager" title="Permanent link">&para;</a></h2>
<p><details style="cursor:pointer;;padding-left:2%">
<summary style="color:#007acc;">[展开] Configure /etc/alertmanager/alertmanager.yml</summary>
<code><pre>
sudo mkdir -p /etc/alertmanager

sudo cat <<-'EOF'>/etc/alertmanager/alertmanager.yml
global:
  resolve_timeout: 5m
  smtp_smarthost: 'smtp.163.com:465'
  smtp_from: 'you_email_user@163.com'
  smtp_auth_username: 'you_email_user@163.com'
  smtp_auth_password: 'you_email_password'
  smtp_require_tls: false # see: smtp 454 Command not permitted when TLS active
  smtp_hello: '163.com'
  #wechat_api_url: 'https://qyapi.weixin.qq.com/cgi-bin/'

templates:
  - 'template/*.tmpl'

route:
  group_by: ['alertname'] # 报警分组依据
  group_wait: 10s # 最初即第一次等待多久时间发送一组警报的通知
  group_interval: 10s # 在发送新警报前的等待时间
  repeat_interval: 1m # 发送重复警报的周期, 对于email此项不可以设置过低, 以防smtp服务器拒绝
  receiver: 'email' # 对应以下receivers

receivers:
  - name: 'email'
    email_configs:
      - to: '983708408@qq.com'
        html: '{{ template "email.tmpl" . }}'
        headers: { Subject: "[WARN] 报警邮件"}
  #- name: 'webhook'
  #  webhook_configs:
  #    - url: 'http://127.0.0.1:5001'
  #      send_resolved: true
  #- name: 'wechat'
  #  wechat_configs:
  #    - send_resolved: true
  #      to_party: '1' # 接收组ID
  #      agent_id: '1000002' # 企业微信->自定应用->AgentId
  #      corp_id: '******' # 企业信息(我的企业->CorpId[在底部])
  #      api_secret: '******' # 企业微信(企业微信->自定应用->Secret)
  #      message: '{{ template "test_wechat.html" . }}'

# 抑制器配置
inhibit_rules:
  - source_match: # 源标签警报触发时抑制含有目标标签的警报，在当前警报匹配 status: 'High'
      severity: 'critical'
      status: 'High'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'dev', 'instance']
EOF
</pre></code>
</details></p>

<ul>
<li>Configure email.tmpl</li>
</ul>
<div class="highlight"><pre><span></span><code><a id="__codelineno-1-1" name="__codelineno-1-1"></a><a href="#__codelineno-1-1"><span class="linenos" data-linenos=" 1 "></span></a>sudo cat <span class="s">&lt;&lt;-&#39;EOF&#39;&gt;/etc/alertmanager/template/email.tmpl</span>
<a id="__codelineno-1-2" name="__codelineno-1-2"></a><a href="#__codelineno-1-2"><span class="linenos" data-linenos=" 2 "></span></a><span class="s">{{ define &quot;email.html&quot; }}</span>
<a id="__codelineno-1-3" name="__codelineno-1-3"></a><a href="#__codelineno-1-3"><span class="linenos" data-linenos=" 3 "></span></a><span class="s">&lt;table border=&quot;1&quot;&gt;</span>
<a id="__codelineno-1-4" name="__codelineno-1-4"></a><a href="#__codelineno-1-4"><span class="linenos" data-linenos=" 4 "></span></a><span class="s">        &lt;tr&gt;</span>
<a id="__codelineno-1-5" name="__codelineno-1-5"></a><a href="#__codelineno-1-5"><span class="linenos" data-linenos=" 5 "></span></a><span class="s">                &lt;td&gt;报警项&lt;/td&gt;</span>
<a id="__codelineno-1-6" name="__codelineno-1-6"></a><a href="#__codelineno-1-6"><span class="linenos" data-linenos=" 6 "></span></a><span class="s">                &lt;td&gt;实例&lt;/td&gt;</span>
<a id="__codelineno-1-7" name="__codelineno-1-7"></a><a href="#__codelineno-1-7"><span class="linenos" data-linenos=" 7 "></span></a><span class="s">                &lt;td&gt;报警阀值&lt;/td&gt;</span>
<a id="__codelineno-1-8" name="__codelineno-1-8"></a><a href="#__codelineno-1-8"><span class="linenos" data-linenos=" 8 "></span></a><span class="s">                &lt;td&gt;开始时间&lt;/td&gt;</span>
<a id="__codelineno-1-9" name="__codelineno-1-9"></a><a href="#__codelineno-1-9"><span class="linenos" data-linenos=" 9 "></span></a><span class="s">        &lt;/tr&gt;</span>
<a id="__codelineno-1-10" name="__codelineno-1-10"></a><a href="#__codelineno-1-10"><span class="linenos" data-linenos="10 "></span></a><span class="s">        {{ range $i, $alert := .Alerts }}</span>
<a id="__codelineno-1-11" name="__codelineno-1-11"></a><a href="#__codelineno-1-11"><span class="linenos" data-linenos="11 "></span></a><span class="s">                &lt;tr&gt;</span>
<a id="__codelineno-1-12" name="__codelineno-1-12"></a><a href="#__codelineno-1-12"><span class="linenos" data-linenos="12 "></span></a><span class="s">                        &lt;td&gt;{{ index $alert.Labels &quot;alertname&quot; }}&lt;/td&gt;</span>
<a id="__codelineno-1-13" name="__codelineno-1-13"></a><a href="#__codelineno-1-13"><span class="linenos" data-linenos="13 "></span></a><span class="s">                        &lt;td&gt;{{ index $alert.Labels &quot;instance&quot; }}&lt;/td&gt;</span>
<a id="__codelineno-1-14" name="__codelineno-1-14"></a><a href="#__codelineno-1-14"><span class="linenos" data-linenos="14 "></span></a><span class="s">                        &lt;td&gt;{{ index $alert.Annotations &quot;value&quot; }}&lt;/td&gt;</span>
<a id="__codelineno-1-15" name="__codelineno-1-15"></a><a href="#__codelineno-1-15"><span class="linenos" data-linenos="15 "></span></a><span class="s">                        &lt;td&gt;{{ $alert.StartsAt }}&lt;/td&gt;</span>
<a id="__codelineno-1-16" name="__codelineno-1-16"></a><a href="#__codelineno-1-16"><span class="linenos" data-linenos="16 "></span></a><span class="s">                &lt;/tr&gt;</span>
<a id="__codelineno-1-17" name="__codelineno-1-17"></a><a href="#__codelineno-1-17"><span class="linenos" data-linenos="17 "></span></a><span class="s">        {{ end }}</span>
<a id="__codelineno-1-18" name="__codelineno-1-18"></a><a href="#__codelineno-1-18"><span class="linenos" data-linenos="18 "></span></a><span class="s">&lt;/table&gt;</span>
<a id="__codelineno-1-19" name="__codelineno-1-19"></a><a href="#__codelineno-1-19"><span class="linenos" data-linenos="19 "></span></a><span class="s">{{ end }}</span>
<a id="__codelineno-1-20" name="__codelineno-1-20"></a><a href="#__codelineno-1-20"><span class="linenos" data-linenos="20 "></span></a><span class="s">EOF</span>
</code></pre></div>
<ul>
<li>Deployment alertmanager server</li>
</ul>
<div class="highlight"><pre><span></span><code><a id="__codelineno-2-1" name="__codelineno-2-1"></a><a href="#__codelineno-2-1"><span class="linenos" data-linenos="1 "></span></a><span class="c1"># @see: https://hub.docker.com/r/prom/alertmanager/tags</span>
<a id="__codelineno-2-2" name="__codelineno-2-2"></a><a href="#__codelineno-2-2"><span class="linenos" data-linenos="2 "></span></a>docker pull docker.io/prom/alertmanager:v0.23.0
<a id="__codelineno-2-3" name="__codelineno-2-3"></a><a href="#__codelineno-2-3"><span class="linenos" data-linenos="3 "></span></a>
<a id="__codelineno-2-4" name="__codelineno-2-4"></a><a href="#__codelineno-2-4"><span class="linenos" data-linenos="4 "></span></a>docker run -d --name<span class="o">=</span>alertmanager1 -p <span class="m">9093</span>:9093 -v /etc/alertmanager/:/etc/alertmanager/ -v /mnt/disk1/alertmanager/:/alertmanager/ --network host --restart<span class="o">=</span>always prom/alertmanager:v0.23.0
</code></pre></div>
<ul>
<li>
<p>控制台界面: http://10.111.178.62:9093/</p>
</li>
<li>
<p>AlertManager 效果图</p>
</li>
</ul>
<p><img alt="" src="https://blogs.wl4g.com:443/wp-content/uploads/2021/10/prom-grafana1-3.png" /></p>
<h2 id="5-faq">5. FAQ<a class="headerlink" href="#5-faq" title="Permanent link">&para;</a></h2>
<h3 id="51-prometheus">5.1 启动 prometheus 容器报错？<a class="headerlink" href="#51-prometheus" title="Permanent link">&para;</a></h3>
<ul>
<li>
<p>ERROR: <code>level=error ts=2021-10-24T13:08:47.353Z caller=query_logger.go:87 component=activeQueryTracker msg="Error opening query log file" file=/prometheus/queries.active err="open /prometheus/queries.active: permission denied" panic: Unable to create mmap-ed active query log</code></p>
</li>
<li>
<p>原因: 由于 prometheus 官方镜像启动是用的 nobody 用户, 而在宿主机上启动容器是root, 因此无权限，简单解决方法是设置挂载的宿主上的目录(/mnt/disk1/prometheus)为 777.</p>
</li>
<li><a href="https://blog.51cto.com/u_14268033/2462243">参考: Prometheus 监控 k8s 集群使用邮箱和微信告警</a></li>
</ul>
<h3 id="52-prometheus">5.2 如何清理prometheus 时序数据？<a class="headerlink" href="#52-prometheus" title="Permanent link">&para;</a></h3>
<ul>
<li>
<p>官方文档: https://prometheus.io/docs/prometheus/latest/querying/api/#delete-series</p>
</li>
<li>
<p>首先确保 prometheus 启动参数指定了 <code>--web.enable-admin-api</code>，例如:</p>
</li>
</ul>
<div class="highlight"><pre><span></span><code><a id="__codelineno-3-1" name="__codelineno-3-1"></a><a href="#__codelineno-3-1"><span class="linenos" data-linenos="1 "></span></a>./prometheus --storage.tsdb.retention<span class="o">=</span>180d --web.enable-admin-api
</code></pre></div>
<ul>
<li>删除 mysql 实例: <code>rds-node1:3066</code> 的指标: <code>mysql_global_status_threads_running</code> 的<strong><em>所有数据</em></strong></li>
</ul>
<div class="highlight"><pre><span></span><code><a id="__codelineno-4-1" name="__codelineno-4-1"></a><a href="#__codelineno-4-1"><span class="linenos" data-linenos="1 "></span></a>curl -X POST -g <span class="s1">&#39;http://10.111.178.62:9090/api/v1/admin/tsdb/delete_series?match[]=up&amp;match[]=mysql_global_status_threads_running{instance=&quot;rds-node1:3066&quot;,job=&quot;mysql&quot;}&#39;</span>
</code></pre></div>
<ul>
<li>删除 mysql 实例: <code>rds-node1:3066</code> 的指标: <code>mysql_global_status_threads_running</code> 在时间区间 <strong><em>1557903714</em></strong> 到 <strong><em>155790395</em></strong> 的数据</li>
</ul>
<div class="highlight"><pre><span></span><code><a id="__codelineno-5-1" name="__codelineno-5-1"></a><a href="#__codelineno-5-1"><span class="linenos" data-linenos="1 "></span></a>curl -X POST -g <span class="s1">&#39;http://localhost:9090/api/v1/admin/tsdb/delete_series?start=1557903714&amp;end=1557903954&amp;match[]=up&amp;match[]=mysql_global_status_threads_running{instance=&quot;rds-node1:3066&quot;,job=&quot;mysql&quot;}&#39;</span>
</code></pre></div>
<ul>
<li>删除 mysql 实例: <code>rds-node1:3066</code> 的<strong><em>所有指标</em></strong>在时间区间 <strong><em>1557903714</em></strong> 到 <strong><em>155790395</em></strong> 的数据</li>
</ul>
<div class="highlight"><pre><span></span><code><a id="__codelineno-6-1" name="__codelineno-6-1"></a><a href="#__codelineno-6-1"><span class="linenos" data-linenos="1 "></span></a>curl -X POST -g <span class="s1">&#39;http://localhost:9090/api/v1/admin/tsdb/delete_series?start=1557903714&amp;end=1557903954&amp;match[]=up&amp;match[]={instance=&quot;rds-node1:3066&quot;,job=&quot;mysql&quot;}&#39;</span>
</code></pre></div>
<h3 id="53-grafana">5.3 如何扩展自定义 Grafana 仪表盘？<a class="headerlink" href="#53-grafana" title="Permanent link">&para;</a></h3>
<div class="highlight"><pre><span></span><code><a id="__codelineno-7-1" name="__codelineno-7-1"></a><a href="#__codelineno-7-1"><span class="linenos" data-linenos=" 1 "></span></a>-&gt; 左侧主菜单(Dashboard)
<a id="__codelineno-7-2" name="__codelineno-7-2"></a><a href="#__codelineno-7-2"><span class="linenos" data-linenos=" 2 "></span></a>-&gt; Manage
<a id="__codelineno-7-3" name="__codelineno-7-3"></a><a href="#__codelineno-7-3"><span class="linenos" data-linenos=" 3 "></span></a>-&gt; 选择现有仪表板(进入)
<a id="__codelineno-7-4" name="__codelineno-7-4"></a><a href="#__codelineno-7-4"><span class="linenos" data-linenos=" 4 "></span></a>-&gt; Dashboard settings(一般导入的开源模版默认是只读的) 
<a id="__codelineno-7-5" name="__codelineno-7-5"></a><a href="#__codelineno-7-5"><span class="linenos" data-linenos=" 5 "></span></a>-&gt; Make Editable
<a id="__codelineno-7-6" name="__codelineno-7-6"></a><a href="#__codelineno-7-6"><span class="linenos" data-linenos=" 6 "></span></a>-&gt; 左上角返回 (此时显示的整个仪表盘是可编辑状态) 
<a id="__codelineno-7-7" name="__codelineno-7-7"></a><a href="#__codelineno-7-7"><span class="linenos" data-linenos=" 7 "></span></a>-&gt; Add panel (右上角)
<a id="__codelineno-7-8" name="__codelineno-7-8"></a><a href="#__codelineno-7-8"><span class="linenos" data-linenos=" 8 "></span></a>-&gt; Add an empty panel
<a id="__codelineno-7-9" name="__codelineno-7-9"></a><a href="#__codelineno-7-9"><span class="linenos" data-linenos=" 9 "></span></a>-&gt; 在 Metrics browser 输入要展示的指标expr (如: node_memory_MemFree_bytes{job=&quot;node&quot;,instance=&quot;$instance&quot;})
<a id="__codelineno-7-10" name="__codelineno-7-10"></a><a href="#__codelineno-7-10"><span class="linenos" data-linenos="10 "></span></a>-&gt; Legend (设置显示别名)
<a id="__codelineno-7-11" name="__codelineno-7-11"></a><a href="#__codelineno-7-11"><span class="linenos" data-linenos="11 "></span></a>-&gt; 右上角 search by 选择图表类型，常用 Graph (old)
<a id="__codelineno-7-12" name="__codelineno-7-12"></a><a href="#__codelineno-7-12"><span class="linenos" data-linenos="12 "></span></a>-&gt; Apply (预览)
</code></pre></div>
<h3 id="54-grafana">5.4 分享自定义的表盘到 Grafana 官方，上传时报错？<a class="headerlink" href="#54-grafana" title="Permanent link">&para;</a></h3>
<ul>
<li>
<p>ERROR; <code>Dashboards from Grafana v3.0 or earlier cannot be uploaded.</code></p>
</li>
<li>
<p>Grafana 3.1 前后导出的仪表盘 json 格式存在差异，大于 3.1 增加了 <code>__inputs</code> 前缀，导出时必须勾选 <code>Export for sharing externally</code>，当然导入新版仪表盘到本地低版本是有兼容的，见官方说明: https://grafana.com/docs/grafana/v7.5/dashboards/export-import/#note</p>
</li>
<li>
<p>分享上传步骤:</p>
</li>
<li>a. 首先进入当前账号主页: https://grafana.com/auth/sign-in?plcmt=top-nav&amp;cta=myaccount</li>
<li>b. 依次进入菜单: ORG SETTINGS -&gt; My Dashboards -&gt; Upload Dashboards</li>
</ul>
<h3 id="55-grafana">5.5 grafana 管理员账号密码忘记如何重置？<a class="headerlink" href="#55-grafana" title="Permanent link">&para;</a></h3>
<ul>
<li>方式1：</li>
</ul>
<div class="highlight"><pre><span></span><code><a id="__codelineno-8-1" name="__codelineno-8-1"></a><a href="#__codelineno-8-1"><span class="linenos" data-linenos="1 "></span></a>docker <span class="nb">exec</span> -it grafana1 /bin/sh
<a id="__codelineno-8-2" name="__codelineno-8-2"></a><a href="#__codelineno-8-2"><span class="linenos" data-linenos="2 "></span></a>grafana-cli admin reset-admin-password <span class="m">123456</span>
</code></pre></div>
<ul>
<li>方式2：</li>
</ul>
<div class="highlight"><pre><span></span><code><a id="__codelineno-9-1" name="__codelineno-9-1"></a><a href="#__codelineno-9-1"><span class="linenos" data-linenos="1 "></span></a><span class="c1"># 找到 sqllite db 文件</span>
<a id="__codelineno-9-2" name="__codelineno-9-2"></a><a href="#__codelineno-9-2"><span class="linenos" data-linenos="2 "></span></a>find / -name grafana.db
<a id="__codelineno-9-3" name="__codelineno-9-3"></a><a href="#__codelineno-9-3"><span class="linenos" data-linenos="3 "></span></a>sqlite3 /var/lib/grafana/grafana.db
<a id="__codelineno-9-4" name="__codelineno-9-4"></a><a href="#__codelineno-9-4"><span class="linenos" data-linenos="4 "></span></a><span class="c1"># 查看user表</span>
<a id="__codelineno-9-5" name="__codelineno-9-5"></a><a href="#__codelineno-9-5"><span class="linenos" data-linenos="5 "></span></a><span class="k">select</span> * from user<span class="p">;</span>
<a id="__codelineno-9-6" name="__codelineno-9-6"></a><a href="#__codelineno-9-6"><span class="linenos" data-linenos="6 "></span></a><span class="c1"># 更改密码为 admin</span>
<a id="__codelineno-9-7" name="__codelineno-9-7"></a><a href="#__codelineno-9-7"><span class="linenos" data-linenos="7 "></span></a>update user <span class="nb">set</span> <span class="nv">password</span> <span class="o">=</span> <span class="s1">&#39;59acf18b94d7eb0694c61e60ce44c110c7a683ac6a8f09580d626f90f4a242000746579358d77dd9e570e83fa24faa88a8a6&#39;</span>, <span class="nv">salt</span> <span class="o">=</span> <span class="s1">&#39;F3FAxVm33R&#39;</span> where <span class="nv">login</span> <span class="o">=</span> <span class="s1">&#39;admin&#39;</span><span class="p">;</span>
</code></pre></div>
<h2 id="6">6. 参考<a class="headerlink" href="#6" title="Permanent link">&para;</a></h2>
<ul>
<li><a href="https://github.com/prometheus/node_exporter#enabled-by-default">参考 1: node_exporter 默认启用的指标 (https://github.com/prometheus/node_exporter#enabled-by-default)</a></li>
<li><a href="https://www.jianshu.com/p/3b08a8dea9e5">参考 2: node_exporter 指标扩展分析</a></li>
<li><a href="https://www.infoq.cn/article/Prometheus-theory-source-code">参考 3: Prometheus client_golang 库源码分析</a></li>
<li><a href="https://blog.csdn.net/alex_yangchuansheng/article/details/111087619">参考 4: 一键备份导出生产环境所有 Grafana dashboard</a></li>
<li><a href="http://t.zoukankan.com/rongfengliang-p-13518987.html">参考 5: Grafana dashboard 分享嵌入到外部系统的方式</a></li>
<li><a href="https://zuozewei.blog.csdn.net/article/details/115058528">参考 6: Prometheus + node_exporter + Grafana 实现主机监控</a></li>
<li><a href="https://www.jianshu.com/p/1697e09e99ac">参考 7: Prometheus (rule)+ alertmanager 监控告警整合</a></li>
<li><a href="https://blog.51cto.com/ylw6006/2084403">参考 8: Kubernetes 集群 Prometheus + Grafana 监控方案</a></li>
</ul>

              
            </article>
          </div>
        </div>
        
      </main>
      
        <footer class="md-footer">
  
  <div class="md-footer-meta md-typeset">
    <div class="md-footer-meta__inner md-grid">
      <div class="md-copyright">
  
  
    Made with
    <a href="https://squidfunk.github.io/mkdocs-material/" target="_blank" rel="noopener">
      Material for MkDocs
    </a>
  
</div>
      
    </div>
  </div>
</footer>
      
    </div>
    <div class="md-dialog" data-md-component="dialog">
      <div class="md-dialog__inner md-typeset"></div>
    </div>
    <script id="__config" type="application/json">{"base": "../../../..", "features": ["search.suggest", "search.highlight", "navigation.tabs", "navigation.expand", "toc.follow", "toc.integrate"], "search": "../../../../assets/javascripts/workers/search.5e67fbfe.min.js", "translations": {"clipboard.copied": "Copied to clipboard", "clipboard.copy": "Copy to clipboard", "search.config.lang": "en", "search.config.pipeline": "trimmer, stopWordFilter", "search.config.separator": "[\\s\\-]+", "search.placeholder": "Search", "search.result.more.one": "1 more on this page", "search.result.more.other": "# more on this page", "search.result.none": "No matching documents", "search.result.one": "1 matching document", "search.result.other": "# matching documents", "search.result.placeholder": "Type to start searching", "search.result.term.missing": "Missing", "select.version.title": "Select version"}, "version": {"default": "latest", "provider": "mike"}}</script>
    
    
      <script src="../../../../assets/javascripts/bundle.c44cc438.min.js"></script>
      
        <script src="../../../../static/js/util.js"></script>
      
    
  </body>
</html>